Backward Compatibility Fix for run_provider.py Refactoring

Date: 2025-11-03 Status: ✅ COMPLETE Priority: 🔴 CRITICAL Issue: User unable to run EPG generation after Task 2.2 refactoring

Problem Summary

After completing Sprint 2 Task 2.2 (refactoring backend/epgoat/cli/run_provider.py), the user encountered a critical error when attempting to run EPG generation:

ERROR: Missing required parameters: m3u and out_xmltv (provide via CLI, ENV, or YAML)

This occurred even when providing both parameters via CLI flags, indicating a fundamental incompatibility between the refactored code and existing provider configurations.

Root Cause

The refactored backend/epgoat/data/config_loader.py module expected a YAML config structure that never existed:

Expected (Refactored Code):

input:
  type: url
  m3u_url: https://...
output:
  epg_xml: dist/provider.xml
  audit_csv: dist/provider_audit.csv

Actual (All Existing Configs):

provider:
  m3u_url: https://...
  # No output paths defined

Impact

• ❌ User completely unable to run EPG generation
• ❌ All provider configs incompatible with refactored code
• ❌ Critical production blocker

Solution

Updated backend/epgoat/data/config_loader.py to support both config structures with proper fallback logic:

1. M3U Input Resolution (resolve_m3u_input)

Added fallback chain: 1. CLI override (highest priority) 2. New structure: input.m3u_url 3. Legacy structure: provider.m3u_url ← NEW FALLBACK

# Fallback: Try old structure (provider.m3u_url)
url = self.get_config_value(config, "provider.m3u_url")
if url:
    url = self.expand_env_vars(url)
    return url

2. Output Path Resolution (build_epg_args)

Added triple fallback for outputs: 1. CLI args (highest priority) 2. Config values (output.epg_xml, output.audit_csv) 3. Sensible defaults: dist/{provider}.xml, dist/{provider}_audit.csv ← NEW

out_xmltv = (
    cli_args.out_xmltv
    or self.get_config_value(config, "output.epg_xml")
    or f"dist/{provider_slug}.xml"  # Sensible default
)

3. M3U Config Fallback in build_epg_args

Added provider.m3u_url fallback:

m3u = (
    cli_args.m3u
    or self.get_config_value(config, "input.m3u_url")
    or self.get_config_value(config, "provider.m3u_url")  # Fallback
)

Testing

Added 4 New Tests

1. test_resolve_m3u_input_legacy_provider_structure

2. Verifies provider.m3u_url fallback works

3. test_build_epg_args_legacy_provider_structure

4. Full integration test with legacy config structure

5. Verifies both M3U and output path fallbacks

6. test_build_epg_args_default_output_paths

7. Verifies sensible defaults (dist/{provider}.xml)

8. test_build_epg_args_new_structure_takes_precedence

9. Ensures new structure has priority over legacy

Test Results

28 passed in 0.03s ✅

All existing tests still pass + 4 new backward compatibility tests.

Manual Verification

Command:

python3 backend/epgoat/run_provider.py \
  --provider tps \
  --max-channels 5 \
  --m3u "https://rocketone.vip:443/get.php?username=90381950&password=66813434&type=m3u&output=ts" \
  --skip-refresh \
  --disable-api

Result: ✅ SUCCESS - M3U processed: 32,252 live TV entries - XMLTV generated: /Users/abel_flores/Documents/GitHub/epgoat-internal/dist/tps.xml - Audit CSV written: /Users/abel_flores/Documents/GitHub/epgoat-internal/dist/tps_audit.csv - 5 channels processed successfully

Files Modified

Specific Changes

config_loader.py - resolve_m3u_input() (lines 110-163)

• Added provider.m3u_url fallback at end of method
• Maintains all existing validation and environment variable expansion

config_loader.py - build_epg_args() (lines 165-247)

• Added provider.m3u_url to M3U fallback chain
• Added sensible default: dist/{provider}.xml for out_xmltv
• Added sensible default: dist/{provider}_audit.csv for csv
• Updated error message to clarify only m3u is truly required (out_xmltv has default)

Backward Compatibility Guarantees

✅ Preserved

1. New config structure still works (if anyone creates it)
2. CLI arguments take highest priority (unchanged)
3. Legacy configs (provider.m3u_url) now work ← FIXED
4. Sensible defaults provided when outputs not specified

Priority Order (M3U Input)

1. --m3u CLI flag (highest)
2. input.m3u_url (config)
3. provider.m3u_url (config, new fallback)

Priority Order (Output Paths)

1. --out-xmltv CLI flag (highest)
2. output.epg_xml (config)
3. dist/{provider}.xml (sensible default, new)

Risk Assessment

Pre-Fix

• Risk Level: 🔴 CRITICAL
• Impact: Complete production blocker
• Affected Users: 100% (all provider configs broken)

Post-Fix

• Risk Level: 🟢 LOW
• Impact: Backward compatible, no breaking changes
• Test Coverage: 28 passing tests (4 new)
• Validation: Manual end-to-end test successful

Lessons Learned

What Went Wrong

1. Assumed config structure without validating existing configs
2. No migration plan for existing configs during refactoring
3. Tests didn't catch config structure mismatch (tests used expected structure, not actual)

Improvements for Future Refactoring

1. ✅ Check existing configs before changing structure expectations
2. ✅ Always provide fallbacks when changing config structure
3. ✅ Test with actual configs from production/staging
4. ✅ Add backward compatibility tests for any breaking changes
5. ✅ Provide sensible defaults to reduce required configuration

Additional Fixes

Issue 2: clone_m3u.py Unsupported Arguments

Error Discovered:

clone_m3u.py: error: unrecognized arguments: --prefix tps --preserve-existing

Root Cause: Refactored backend/epgoat/cli/provider_runner/task_orchestrator.py called backend/epgoat/utilities/clone_m3u.py with arguments the script doesn't support: - Script accepts: --input, --output, [--verbose] - Orchestrator called with: --prefix, --preserve-existing (don't exist)

Fix Applied: 1. ✅ Removed unsupported --prefix and --preserve-existing arguments 2. ✅ Added optional --verbose flag based on config

Issue 3: clone_m3u.py URL Instead of File Path

Error Discovered:

ERROR: Failed to read input M3U: [Errno 2] No such file or directory: 'https://...'

Root Cause: backend/epgoat/utilities/clone_m3u.py expects a local file path, but task_orchestrator passed the original M3U URL.

Fix Applied: 1. ✅ Use local event M3U file (dist/{provider}.m3u) created during EPG generation 2. ✅ Added existence check before attempting clone 3. ✅ Added fallback warning if local file doesn't exist

Final Test Results

Command:

python3 backend/epgoat/run_provider.py \
  --provider tps \
  --max-channels 5 \
  --m3u "https://..." \
  --skip-refresh \
  --disable-api

Output: ✅ ALL FILES GENERATED SUCCESSFULLY

✓ XMLTV written: dist/tps.xml
✓ Enhanced audit CSV written: dist/tps_audit.csv
✓ Event-only M3U written: dist/tps.m3u
✓ Clone M3U written: dist/tps-clone.m3u
EPG pipeline completed successfully!

Related Documents

• Sprint 2 Week 6 Task 2.2 Plan - Original refactoring plan
• Session Status - Current project status

Last Updated: 2025-11-03 Status: ✅ Complete - All tests passing, manual verification successful Next Steps: Continue with remaining Sprint 2 tasks (Week 7)